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Abstract. Relations between Shannon entropy and Renyi entropies of integer 
order are discussed. For any iV-point discrete probability distribution for 
which the Renyi entropies of order two and three are known, we provide an 
lower and an upper bound for the Shannon entropy. The average of both 
bounds provide an explicit extrapolation for this quantity. These results imply 
relations between the von Neumann entropy of a mixed quantum state, its 
linear entropy and traces. 



ver. 2 with corrigendum added, February 17, 2005 
1. Introduction 

We are going to analyze discrete probability distributions x — {x\, ..xn}, which 
consist of non-negative numbers summing to unity. Xi = 1. To characterize 

quantitatively such probability vectors one uses Shannon (information) entropy £Q 

N 

(1) H(x) :— - Xj \nxj 

i=l 

where we adopt the convention that OlnO = 0, if necessary. 

The Shannon entropy is distinguished by several unique properties [2], but it 
is often convenient to introduce generalized Renyi entropies [3] parametrized by a 
continuous parameter q, 

(2) ff 9 (z):=-Lln[f>?]. 

y »=i 

The Renyi entropies are well defined for positive g ^ 1, but is is not difficult to 
show that for any probability distribution x one has lim 9 ^i H q (x) = H{x). For 
consistency the Shannon entropy H will thus be denoted by H\. This method of 
generalizing the Shannon entropy is by far not the only one - for reviews of other 
generalizations see books by Kapur 0] and Arndt 

In this work we discuss relations between Renyi entropies of different orders, 
and in particular between Hi, H2 and H3. Physical motivation for such a study is 
twofold. First, we may not know the entire vector x, but only a few of its L q norms, 
so knowing the Renyi entropies, say H 2 and H 3 we want to estimate the unknown 
Shannon entropy H\. Such a situation occurs if one studies an N dimensional 
quantum mechanical mixed state p according to the scheme recently proposed by 
P. Horodecki et al. [HJ|7] and measures directly the traces Trp fe for k = 2, 3, ...M. 
If M < N the entire spectrum of p remains unknown, and it is not possible to 
find its von Neumann entropy, S(p) = — Trplnp, (i.e. the Shannon entropy of the 
spectrum), but the generalized Renyi entropies H^, including the linear entropy, 
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which is a function of H2 may be readily obtained. Similar problems arise in many 
different branches of physics, for instance by the study of the statistics between 
particles created in high-energy collisions |SJ|H]- Measuring probabilities of that 
two independent collisions give rise to the same distribution of particles allows one 
to obtain the Renyi entropy Hi, but not directly the Shannon entropy H±. 

Another reason to study relations between H± and Hi has been provided by 
the work by Pipek and Varga ^H]- They assumed that both these quantities are 
known, and observed that their difference, S s t r ■= H± — H% called structural en- 
tropy, provides an important characterization of the analyzed probability vector x. 
For instance, an increase of the structural entropy charactering an eigcnstate of a 
tight binding model indicates the Anderson transition. Several other applications 
of structural entropy include also quantum chemistry and statistical analysis of 
quantum spectra, (see and references therein). 

The von Neumann entropy of a mixed state obtained by partial trace of a bi- 
partite pure state, p = Tr^ (l^) measures the degree of entanglement of the 
pure state \tp). Alternatively one can measure the entanglement by generalized en- 
tropies (see e.g. ^) . so relations between entropies analyzed in this work provide 
bounds between different measures of entanglement. This very point has recently 
been discussed in the paper by Wei et al. |13|. which provides an additional moti- 
vation for the present work. 

This paper is ogranized as follows. In section 2 the basic properties of the 
Renyi entropies are reviewed. In section 3 we present recent results of Tops0e 
and Harremoes (141 IT5] , which allow us to propose lower and upper bounds on 
the Shannon entropy obtained out of the Renyi entropies of order two and three, 
provided the length N of the vector is known. They are derived in section 4, while 
in section 5 we propose and analyze en estimation of the Shannon entropy. 

2. Shannon and Renyi entropies 

Consider a random variable £ attaining not more than N different values with 
probabilities X4, i = 1,...,N. Such discrete probability distribution P may be 
cahracterized by the Shannon entropy Q or generalized Renyi entropies PJ . 

All generalized entropies H q vary from zero for a certain event (the distri- 
bution Q\ :— {1,0,. ..,0}) to lniV, for the uniform distribution, (the distribu- 
tion Qm '■= {1/N,1/N, ...,1/N}). For the distributions with k equal elements, 
Qk '■= {1/fe, 1/fc, 0, 0}, the entropies admit intermediate values, In k. 

The Renyi entropy H q converges to the Shannon entropy in the limit q — ► 1. It 
is also useful to express the Shannon entropy as the limit of the derivative, 

(3) J™ dq ■ 

Some special cases of H q are of special interest. For q = 2 we have H^x*) = 
~ m Ei!=i X T\- The Renyi entropy of order two, called extension entropy ^Oh is 
closely related to the inverse participation ratio, 

(4) fl(g) := _i . =exp[ff 2 (£)]. 



i=l 



.).:? 



This quantity characterizes the "effective number of different events" which the 
stochastic variable may admit, and varies from unity for Q\,toN for the uniform 
distribution Qn- Another quantity r = 1/R= J2iLi x h called index of coincidence 
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|15j . in quantum mechanical problems is called purity, since the larger r the more 
pure, the state it describes. The quantity L := 1 — r = 1 — exp(— H 2 ) is called 
linear entropy since in analogy to Shannon entropy it achieves its maximum for the 
uniform distribution Qn- 

In the case q = the Renyi entropy is a function of the number to of positive 
components of the vector, Hq(x) — In to. In the limit q — > 00 we obtain a quantity 
analogous to the Chebyshev norm: = — lnx max , where z max is the largest 

component of x. 

The Renyi entropy is a sum of N terms so for any finite N the function of 
H q on q is differentiable. The functional dependence of the Renyi entropy on its 
parameter was investigated in detail by Back and Schlogl 16 ). Making use of the 
fact that the function x s is convex for s > 1 and concave for < s < 1 they have 
proved several inequalities 1 , which we recall in the case q > 0, 

(5) §- q H q < 0, 



(6) ^r q —H q > 0, 

oq q 



(7) £- q (l-q)H q <0, 

(8) ^{l-q)H q >Q. 

The first inequality @ means that the Renyi entropy is a non increasing function 
of its parameter, 

(9) H q (x) > H s (x) for any s>q 

and this statement is valid also for infinite probability vectors and the cases of 
nondiffcrcntiablc H q |16| . Hence the structural entropy S str := Hi — H 2 is non- 
negative [TU]. 

Inequality (JSJ) implies that the dependence of the Renyi entropy on its parameter 
is convex. 2 Thus knowing the and 2-entropies one obtains by linear interpolation 
an upper bound for the Shannon entropy 

(10) Hi(aO < H u0 := ^(H (x) + H 2 (x)). 
This relation gives us an upper bound for the structural entropy 

(11) S str (x) := H x {x) - H 2 (x) < ^(H (x) - H 2 (x)) 

valid for any vector x of a finite length TV. 

In an analogous way, if the Renyi entropies of order 2 and 3 are known, the linear 
extrapolation provides a lower bound for the Shannon entropy 

(12) H x {x) > H d23 (x) = 2H 2 (x) - H 3 (x), 



^In the book [T51 the quantity — H q called Renyi information was analyzed, so the direction of 
the inequalities derived there is inverted. 

2 This claim is withdrawn - see into corrigendum Sec. |7] in which consequences of this error 
are pointed out. 
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which combined with (|10|l . allows one to write down a simple estimation ifo23 = 
(H d + H u0 )/2, (seeFig3.a), 

(13) Hx{x) « ff 23(x) := J [#(>(*) + 5ff 2 (f) - 2# 3 (z)]- 
Making use of the inequality JHJ we obtain the relation 

(14) 1—?-H q (x) < '^—^-H s (x) for any s > q , 

q s 

which is equivalent to the statement that the L g -norm is a non-increasing function, 
I \x\ \ s < 1 \x\ \q- This result provides another upper bound, H q < q(s — 1)H S / s(q — 1). 
Although it is not applicable for the Shannon entropy, for which q = 1 so the 
inequality becomes trivial, but it gives an usefull bound on H q with g > 1 by the 
limiting value H^, 

(15) H q {x) < -S—H^g). 

In further sections of this work we shall discuss possibilities of finding more 
precise bounds and estimations for the Shannon entropy, provided the dimension 
N of the probability vector is known. 

3. Bounds between Renyi entropies 

For any value q > the generalized entropy H q is equal zero for certain events 
described by the distribution Qi, and achieves its maximum for the uniform distri- 
bution, S(Q N ) = InN. 

To investigate further relations between the Renyi entropies of different order 
we have chosen to analyze the case of N = 3 dimensional vectors x. The space of 
all possible probability vectors, plotted in the the plane X3 = 1 — x\ — X2 forms 
an equilateral triangle of side y/2 measured in the Euclidean distance. Its three 
corners: (100), (010) and (001) represent certain events, while the center of the 
triangle corresponds to the uniform distribution Q3. 

Fig. H shows sets of points characterized by the same Renyi entropy of order q, 
which may be called iso-entropy curves. Independently of the value of the parameter 
q the generalized entropy attains its minimum, H q = 0, at the corners of the 
triangle, while the maximum H q — In 3 is achieved at the point Q3 at the center of 
the triangle. As shown in Fig. the maximum is rather flat for q = 1/4. The case 
shown in this panel resembles the limiting case H , for which the entropy reflects 
the number of events which may occur: it vanish at the corners of the triangle, is 
equal to In 2 at its sides and equals to In 3 for any point inside the triangle. The 
other example, q = 8, presented in Fig. ^i. is similar to the limiting case i?oo, 
for which the iso-entropy curves are perpendicular to the lines joining Q3 with the 
corners. 

Superimposing some of the above pictures on one graph allows one to understand 
further relations between the Renyi entropies. The generalized entropies are corre- 
lated; e.g. for the distributions Qk the entropies are equal to In k independently of 
the value of q. 

The problem, which values the entropy H q may admit, provided H s is given, 
has been solved by Harremoes and Tops0e ^3- F° r an Y distribution P £ they 
proved a simple (but not very sharp) upper bound on Hi by H2, 

(16) H 2 (x) < H x {x) < In N + 1/N - exp(-# 2 (£)) 
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Figure 1 . Iso-entropy curves in the space of probability distribu- 
tions with N = 3. The generalized Renyi entropy is constant along 
the curves plotted for (a) q — 1/4, (b) q — 1 (Shannon entropy), 
(c) q = 2 (Euclidean circles-distance D 2 ), and (d) q = 8. Dotted 
lines form the triangle A(Qi, Q2, Qz)- 

where the lower bound is a special case of ©• Moreover, they showed that the set 
A 9jS of possible probability distributions plotted in the plane H q versus H s is not 
convex (see Fig. 2), and its boundaries are formed of arcs corresponding to the 
interpolating probability distributions 

(17) Qk,l(a) ■= a Qk + (1 - a )Qi with ae[0,l]. 

More precisely, for any probability distribution P consisting of N components and 
arbitrary s > q > the following bounds hold [T5\ 

(18) H q (Q k _ hk (a)) < H q (P) < H q {Q 1>N (a)), 

where a is a function of the known value of H S (P) and the natural number k is 
selected by the inequality \n(k — 1) < H S (P) < Ink. 

The above results, crucial for the main body of this work, are easy to understand. 
Let us discuss the simplest nontrivial case with N = 3. The two dimensional simplex 
of probability distributions may be divided into 6 identical parts, equivalent to 
the triangle A(Qi, Q2, Q3), as shown in Fig. lc. Three sides of the triangle are 
formed of the interpolating distributions Qi,2, Qi,3 and Q2.3 and these distinguished 
probabability distributions are extreme in a sense that they lead to the bounds (11811 . 
The bounds between Hi and H2 for N = 3 are presented in Fig. 2a. To obtain them 
it is sufficient to travel along the sides of the triangle A(Qi,Q 2 , Q3), computing Hi 
and H 2 at each point and to plot the data obtained in the plane Hi versus H 2 . 

More formally, the upper boundary of the set A 9jS consist of one arc derived from 
the family of distributions Qi^i®)] for any value of a we compute H s (a), invert it 
to obtain a(H s ) and plot H q (a(H s )). In the case N = 3 the upper bounds plotted 
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in Fig. 2a, c and e arise from the hypotenuse Qi, Q3 of the triangle A(Qi, Q 2 , Q3) 
from Fig. Q]b and c. 

In the similar way the lower bound may be derived from the distributions 
Qk.k+i( a ) f° r k = l,...N — 1. It consists of (TV — 1) arcs forming an cascade 
H q {Qi^{a)) ~— - H q (Q2,s(a)) ■ ■ ■ -—^ H q (Q n -i,n{o)) ■ Note that the distributions 
Qk are represented in each plot by the points (lnfc,lnfc), which connect the neigh- 
bouring arcs. For N = 3 the lower bound consists of two arcs, corresponding to the 
adjacent sides Qi, Q 2 and Q 2 , Q3 of A(Qi, Q2, Qz)- 

The shape of the set A g-S requires a comment. The N — 1 dimensional simplex - 
the set of all iV-points probability distributions is convex and any of its projections 
onto a plane forms a convex set. However, its image at the plane H q versus H s 
needs not to be convex, since the transformations H q (x) and H s (x) are nonlinear. 
The boundaries of A 9)S are obtained as the image of an appropriately chosen path 
on the boundary of the simplex. In the case considered it is the path Q\ — > Q 2 — > 
■ • • — > Qjy — ► Qi) independently of the values of q and s. Observe that the general 
structure of the set A 9jS does not depend on s. However, the larger difference s — q, 
the larger area of the set: the less information on H q is provided by H s . 



N=3 N=5 




Figure 2. The set of all possible discrete distributions for N = 3 
and iV = 5 at the Renyi entropies plane Hi and H q : q = 2 (a,b); 
q = 3 (c,d) and q — 4 (e,f). Thin dotted lines in each panel 
represent the monotonicity lower bounds (J5J) while bold dotted 
curves in panel (a) and (b) denote upper bound (|16|) . 

Let us emphasize in this point that results presented in J^l do not close the issue 
of finding bounds and relations between different entropies. Results analogous to 
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(I18JI for a more general class of entropy functions were recently obtained by Berry 
and Sanders |17| . A more precise lower bound for Shannon entropy, quadaratic in 
terms of index of coincidence (purity) was found by Tops0e |18| . 



4. TV-DEPENDENT BOUNDS FOR SHANNON ENTROPY 

4.1. Bounds based on H2 and the length A of the probability vector. 
Results (|18|l allow us to obtain bounds for a value of the entropy H q , provided the 
value H s is known. Let us first assume that the entropy H 2 is known and we want 
to extract some information on Hi . We start computing the Renyi entropy of order 
two, 



(19) H 2 {Q 1 , N (a)) = -In 

and invert it to obtain 



(l + (iV-l)a^ + (Ar _ i) (l-a) 2 



A 2 



A 2 



(20) 



Aexp(-ff 2 ) - 1 



A - 1 

In this way we receive sharp upper bounds for q € (0, 2) 



(21) 



1 



H q (P) < — 



hi 



1 + (N -l)a\i 
N 



(N-l)( 



l — a\i 



N 



A)- 



°ln. 1 



l + a(N-l) l + a(N-l) 
- In — 



which for q — ► 1 reduces to 

(22) H X{ P)<H1 2 := { 1- N ... N N ... N 

with a given by (|2U|) . 

To obtain analogous lower bound we find k such that ln(fc — 1) < H 2 < Ink 
and compute x,fc(°))- Also this relation may be easily inverted providing 

a = y fc(fc — 1) exp(— i?2) + 1 — fc. Thus we arrive at a lower bound for the Renyi 
entropy 

H q {P) > In (fc - l)z q + y« 



(23) 



and in particular case, for the Shannon entropy 
(24) H X (P) > Hf 2 := (l-fc)zlnz 

with z = (fc + a — l)/(fc 2 — fc) and y = (1 — o)/fc. 



4.2. Bounds based on Hj, and A. Let us now assume, we know the value of 
the Renyi entropy H3. As in (|19l) we compute Hz{Qi,n{q))i and invert it finding 
a 6 [0,1] as the largest (real) root of the polynomial 

3 A 2 exp(-2iJ 3 ) - 1 



(25) 



W u (a) =a A + a< 



A 



= 0. 



(A - 1)(A- 2) 

Then the upper bound valid for q € (0, 3) is given by the same formula l|21(l with 
a given by the root of Q25fl instead of l|2U|) . For q — > 1 one obtains then the upper 
bound for the Shannon entropy 

(26) i^OP^i^V-a-A) 1 



N 



v 1 



A 



l + a(A-l), l + a(A-l) 
- In - 



A 



with a determined by l|25|l . 
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To get the lower bound we look for kl such that hx(k' — 1) < H 3 < Ink'. 
The relation Hs{Qk'—i,k'{p,)) may be inverted explicitly for k' — 2 providing a — 
^/[4cxp(— 2H 3 ) — l]/3. For k' > 2 a is given by the only root of the polynomial 

(27) W d (a) = a 3 + a 2 ^—^ + ^£^[1 - ^ exp(-2ff 3 )] = 0. 

in the interval [0, 1]. The lower bound for the Renyi entropies has the same form 
as (12311 and gives for the Shannon entropy 

(28) Hi (P) >H? 3 :=(1- k')z' In z' - y' In y' 
with z'=(k' + a- l)/{k'{k' - 1)) and y' = (1 - a)/k'. 

5. Combined extrapolation 

In previous section we obtained two upper bounds for the Shannon entropy: ll'MI) 
stemming from the Renyi entropy H2, and l|12|) obtained from H 3 . The latter is in 
general a worse one 3 , but it allows for a linear extrapolation, which gives 

(29) H up (x ) := 2H& (*) ~ H i3 (*) • 

Our numerical results allow us to advance the following 
Conjecture. 

For any probability distribution x the bound 

(30) Hi(x) < H up (x) 
holds. 

In the same way one may try to extrapolate lower bounds defining H d {x) := 
2Hf 2 (x) — Hf 3 (x). For certain probability vectors this quantity may give a useful 
approximation for the Shannon entropy. Interestingly, a relation analogous to (|30|l . 
Hi(x) > Hd(x) is not true: it is violated e.g. if k ^ k'. 

Making use of the rigorous bound l|12|l and the conjecture (|30f) we may suggest 
to estimate the unknown value of the Shannon entropy by the mean value 

(31) Hl{x) := \[H up {x) + H d23 (x)] = H? 2 (x) + H 2 (x) - \[H^{x) + H 3 (x)}, 

which is obtained by combined methods based as well on the lower as well as on 
the upper bounds. Observe that this explicitly computable quantity involves only 
Renyi entropies H 2 and H 3 and the dimension N. This estimation may be improved 
noting that the rigorous lower bounds (|12|) and 124|) are not equivalent. Since for 
some distributions the latter bound gives better (higher) results, we may improve 
(|3*T|l writing 

(32) H*(x) := \h u {x) + imax{ff d23 (x), H( 2 (x)}. 

If the value of the zero-entropy, Hq, is known, one may replace the upper bound 
H u (x) used above by the minimum, min{H u o(x), H up (x)}. 

Figure 3 shows the bounds and the estimations described above for a randomly 
chosen probability distributions with N — 15 components. The overall quality of 
the proposed estimations for the Shannon entropy may be judged from Fig. 4, which 
shows the histogram of the deviations Si — — Hi and ^2 = H d — Hi for a sample 



Knowing the function H{c[) at q = 3 we have less information on Hi, than knowing it at 
g = 2. 
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Figure 3. Renyi entropies H q for a random probability distribu- 
tion of size N = 15 (solid line). Dense dotted curves represent H2 
bounds: upper and lower J33J), while faint dotted curve denote 
analogous bounds based on H3. Straight dashed-dotted lines show 
lower llll'l) and upper (|29|) extrapolations. The exact value of Hi 
is denoted by (*), while the estimation (|32|1 by the middle (x). 



of 10 4 probability vectors generated randomly according to the statistical (Fisher- 
Rao) measure on the N — 1 dimensional simplex ^] . This measure has a simple 
geometric interpretation: is suffices to consider a unit vector t distributed uniformly 
on the sphere S' Ar_1 , and to define the probability vector x — {t\, t 2 N }. Numerical 
results obtained in this way allow us to conclude that the proposed estimation l|32|) 
provides a useful all-purpose approximation of the Shannon entropy. Note that 
the precision of this approximation decreases with the length N of the probability 
vector. 

To judge about possible application of the estimate in the analysis of physical 
data, one should perform analogous numerical simulations with random vectors 
x generated according to a specific probability distribution adjusted to a given 
physical problem. 



6. Concluding Remarks 

In this work we considered the problem of finding the bounds and extrapola- 
tions for the Shannon (entropy), provided some of the Renyi entropies are known. 
In general, generalized entropies of integer order q = 2 and q = 3 are easiest to 
calculate, and they are sufficient to obtain bounds (|10|l and (|12|) for the Shannon 
entropy. The quality of the bound may be improved if the length N of the prob- 
ability vector is known. Then an explicit extrapolation (|32|l allows us to estimate 
the actual value of the entropy H. 
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Figure 4. Histogram of deviations of the extrapolations form the 
real value of the Shannon entropy H±: (*) denotes the probability 
density of error 6% = H* — Hi of the estimation l|32|) . while (o) 
denotes results for the error 6 2 — Hd — Hi of the lower bound (|12|l 
for a sample of 10 4 random probability vectors of size N = 1Q. 

Note that the bounds and extrapolations discussed may be easily rewritten in 
terms of a non-extensive entropy 

1 n 

(33) S q (x):= — 

^ 8=1 

used by Havrda and Charvat [20| and Daroczy |23i which became often used in 
statistical physics after the seminal work of Tsallis [22] ■ In particular, the linear 
entropy L is just the nonextensive entropy of order two, L(x) = S 2 {x), and the 
bounds between H q and Hi imply analogous relations between S q and Hi . In fact 
the plot presented in J2j shows bounds between von Neumann entropy and the 
linear entropy and they follow directly from relation (|18|l proven in JH] ■ 

The issue of comparing the Renyi entropies H , Hi and H 2 is closely related to 
the problem of describing the degree of chaos of an analyzed classical dynamical 
system by the topological entropy Kq, the Kolmogorov-Sinai (metric) entropy Ki 
and the correlation entropy These dynamical entropies are defined as the rate 
of the increase of the Renyi entropies in time but since the length of the 
probability vector is not finite the iV-dependent bounds discussed in this work are 
not applicable. The same concerns comparison of generalized fractal dimensions D q 
of fractal measures: the box-counting dimension Dq, the information dimension Di 
and the correlation dimension D 2 , which also form a non-increasing function of the 
Renyi parameter q [16| . are defined by the limit N — > 00. 

The generalized entropies may also be used to characterize localization properties 
of continuous probability distributions. For instance, any pure quantum state may 
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be represented in the phase space by the Husimi distribution. Its localization 
can be measured by the Wehrl entropy defined as the continuous (Boltzmann- 
Gibbs) entropy of the Husimi distribution |23| . In an analogous way one may 
define generalized Renyi- Wehrl entropies |241 ITT] , and above results may be used 
to obtain similar bounds for the Wehrl entropy. 

It is a pleasure to thank P. Harremoes and F. Tops0e for explaining us their 
results prior to publication and several fruitful comments. I am also thankful to 
I. Bengtsson, A.Bialas, A. Ostruszka, F. Mintert, W. Munro and W. Slomczyhski 
for inspiring discussions and D. Berry, B. Sanders and I. Varga for helpful corre- 
spondence. Financial support by Komitct Badah Naukowych in Warsaw under the 
grant 2P03B-072 19 and by a research grant of the Volkswagen Stiftung is gratefully 
acknowledged. 

7. Corrigendum February 17, 2005 

Eq. JHJ implies that (1 — q)H q is a convex function of q. However, it does 
not imply that the Renyi entropy H q is a convex function of q. For instance, the 
Renyi entropy for a N = 20 probability vector P = (43, 3, . . . , 3) is not a convex 
function of q. I am deeply obliged to Christian Schaffher for drawing my attention 
to this fact. 

Therefore, equations (|ll)p. Q11JI- and l|12fl are not satisfied and conjecture l|3(J|) 
cannot hold. Furthermore, equations |T3J, (|3"TJl . may only be used to extrapo- 
late the unknown value of the Shannon entropy, from the available data on H2 and 
H 3 . 

On the other hand, we would like to mention that the lack of convexity of H q 
in q does not influence the bounds between particular values of the Renyi entropy 
presented in sections 3 and 4 of the present paper. 
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